NPS-AM-14-C1 1 P05R01-031 


PROCEEDINGS 

OF  THE 

ELEVENTH  ANNUAL  ACQUISITION 
RESEARCH  SYMPOSIUM 


WEDNESDAY  SESSIONS 
VOLUME  I 


Identifying  Governance  Best  Practices  in  Systems-of- 

Systems  Acquisition 

Joshua  Archer,  Center  for  Strategic  &  International  Studies 
David  Berteau,  Center  for  Strategic  &  International  Studies 

Published  April  30,  2014 


Approved  for  public  release;  distribution  is  unlimited. 
Prepared  for  the  Naval  Postgraduate  School,  Monterey,  CA  93943. 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


Report  Documentation  Page 

Form  Approved 

OMB  No.  0704-0188 

Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington 

VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  a  penalty  for  failing  to  comply  with  a  collection  of  information  if  it 
does  not  display  a  currently  valid  OMB  control  number. 

1.  REPORT  DATE 

30  APR  2014  2- REPORT  TYPE 

3.  DATES  COVERED 

00-00-2014  to  00-00-2014 

4.  TITLE  AND  SUBTITLE 

Identifying  Governance  Best  Practices  in  Systems-of-Systems  Acquisition 

5a.  CONTRACT  NUMBER 

5b.  GRANT  NUMBER 

5c.  PROGRAM  ELEMENT  NUMBER 

6.  AUTHOR(S) 

5d.  PROJECT  NUMBER 

5e.  TASK  NUMBER 

5f.  WORK  UNIT  NUMBER 

7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

Center  for  Strategic  &  International  Studies, 1616  Rhode  Island  Avenue, 
NW, Washington, DC, 20036 

8.  PERFORMING  ORGANIZATION 

REPORT  NUMBER 

9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 

10.  SPONSOR/MONITOR'S  ACRONYM(S) 

11.  SPONSOR/MONITOR'S  REPORT 
NUMBER(S) 

12.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Approved  for  public  release;  distribution  unlimited 

13.  SUPPLEMENTARY  NOTES 


14.  ABSTRACT 

the  need  for  a  new  model  of  acquisition  governance  ever  more  obvious.  In  order  to  make  better  use  of  its 
defense  dollars  while  contracting  to  acquire  more  advanced  defense  capabilities,  the  defense  acquisition 
community  in  the  United  States  needs  to  improve  governance  processes  for  complex  acquisitions.  Over  the 
last  decade,  researchers  at  the  Center  for  Strategic  and  International  Studies  (CSIS)  have  studied  and 
analyzed  the  increased  challenges  of  complexity  in  defense  acquisitions.  Through  workshops,  conferences, 
and  publications,  CSIS  has  examined  how  the  government  purchases  complex  systems,  what  challenges  it 
encounters  in  doing  so,  and  what  key  attributes  of  governance  can  be  modulated  to  provide  more  effective 
acquisition.  The  study  underlying  this  report  explored  one  simple  question:  In  complex  systems  of  systems, 
what  best  practices  contribute  to  better,  more  efficient  acquisitions?  To  answer  this  question,  the  CSIS 
project  team  reviewed  its  past  work  on  complexity  in  acquisitions  analyzed  new  scholarship  on  the  subject, 
and  conducted  detailed  interviews  with  executives  from  across  the  defense  acquisition  community.  This 
12-month  effort  culminated  with  the  production  of  a  new  model  of  acquisition  governance  model  presented 
in  this  abridged  version  of  a  longer  technical  report  on  the  research.  The  governance  framework  starts 
with  eight  attributes  of  program  management,  which  CSIS  developed  under  its  past  research  on  the 
subject.  The  study  applied  this  framework  to  seven  acquisition  case  examples  of  various  sizes  and 
complexity  which  have  shown  differing  degrees  of  program  success.  CSIS  also  conducted  11  interviews 
with  17  acquisition  executives  and  stakeholders  covering  each  of  the  case  examples  to  help  identify  success 
drivers  and  good  governance  themes.  Finally,  CSIS  used  the  case  studies  and  interviews  to  develop  a  model 
of  acquisition  governance  that  could  help  strengthen  the  ability  of  acquisition  leaders  to  more  effectively 
and  efficiently  acquire  complex  systems  of  systems  (SoS).  The  research  illustrates  how  best  practices  in  the 
three  most  critical  attributes???level  of  organizational  focus,  decision-making  authority,  and 
enforcement?  ??can  make  the  difference  between  success  and  failure  of  an  acquisition  effort.  Best  practices 
in  the  remaining  attributes  also  enable  the  efficient  production  and  procurement  of  effective  systems  of 
systems  and  assist  mission  success  in  large,  complex  acquisitions.  Readers  should  refer  to  the  full  report  for 
supporting  evidence  in  the  form  of  case  studies  and  interview  summaries. 


15.  SUBJECT  TERMS 

16.  SECURITY  CLASSIFICATION  OF: 

17.  LIMITATION  OF 
ABSTRACT 

Same  as 
Report  (SAR) 

18.  NUMBER 
OF  PAGES 

20 

19a.  NAME  OF 
RESPONSIBLE  PERSON 

a.  REPORT 

unclassified 

b.  ABSTRACT 

unclassified 

c.  THIS  PAGE 

unclassified 

Standard  Form  298  (Rev.  8-98) 

Prescribed  by  ANSI  Std  Z39-18 


The  research  presented  in  this  report  was  supported  by  the  Acquisition  Research 
Program  of  the  Graduate  School  of  Business  &  Public  Policy  at  the  Naval 
Postgraduate  School. 

To  request  defense  acquisition  research,  to  become  a  research  sponsor,  or  to  print 
additional  copies  of  reports,  please  contact  any  of  the  staff  listed  on  the  Acquisition 
Research  Program  website  (www.acquisitionresearch.net). 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


Panel  5.  Beyond  Better  Buying  Power:  Assessing 
Progress  and  Institutionalizing  Success 


Wednesday,  May  14,  2014 

1:45  p.m.  - 
3:15  p.m. 

Chair:  James  E.  Thomsen,  Principal  Civilian  Deputy,  Assistant  Secretary  of  the 
Navy  (Research,  Development,  &  Acquisition) 

Identifying  Governance  Best  Practices  in  Systems-of-Systems  Acquisition 

Joshua  Archer,  Center  for  Strategic  &  International  Studies 

David  Berteau,  Center  for  Strategic  &  International  Studies 

Beyond  Business  as  Usual?  Better  Buying  Power  and  the  Prospects  for 
Change  in  Defense  Acquisition 

Zach  Huitink,  Syracuse  University 

Creating  a  Cost-Conscious  Culture:  From  RFP  Through  Execution 

Kathlyn  Hopkins,  Marine  Corps  Systems  Command 

MAinurtu  ru  ioimu4. 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-93- 


Identifying  Governance  Best  Practices  in  Systems-of- 

Systems  Acquisition 


Joshua  Archer — is  a  research  associate  with  the  National  Security  Program  on  Industry  and 
Resources  at  CSIS,  where  his  research  interests  include  defense  industrial  supply  chain  economics 
and  international  joint  development  programs,  particularly  between  the  United  States  and  Japan. 
Before  joining  CSIS,  he  performed  research  in  the  defense  and  aerospace  industry  as  a  strategy 
consultant  between  2010  and  2013.  In  2009,  he  was  a  researcher  with  the  James  Martin  Center  for 
Nonproliferation  Studies,  where  he  contributed  to  the  Taiwan  military  capabilities  profile  of  the 
Nuclear  Threat  Initiative  database.  Archer  holds  an  MA  in  international  policy  studies  from  the 
Monterey  Institute  of  International  Studies  and  is  fluent  in  Japanese. 

David  Berteau — is  senior  vice  president  and  director  of  the  CSIS  International  Security  Program, 
covering  defense  management,  programs,  contracting,  and  acquisition.  His  group  also  assesses 
national  security  economics  and  industry.  Berteau  is  an  adjunct  professor  at  Georgetown  University, 
a  director  of  the  Procurement  Round  Table,  and  a  fellow  of  the  National  Academy  of  Public 
Administration  and  the  Robert  S.  Strauss  Center  at  the  University  of  Texas.  Prior  to  joining  CSIS,  he 
was  director  of  national  defense  and  homeland  security  for  Clark  &  Weinstock,  director  of  Syracuse 
University’s  National  Security  Studies  Program,  and  a  senior  vice  president  at  Science  Applications 
International  Corporation  (SAIC).  He  served  in  the  U.S.  Defense  Department  under  four  defense 
secretaries,  including  four  years  as  principal  deputy  assistant  secretary  of  defense  for  production  and 
logistics.  Berteau  graduated  with  a  BA  from  Tulane  University  in  1971  and  received  his  master’s 
degree  in  1981  from  the  LBJ  School  of  Public  Affairs  at  the  University  of  Texas,  [dberteau@csis.org] 

Abstract 

Increasing  budgetary  constraints  and  the  rapid  growth  of  threats  to  global  security  have  made 
the  need  for  a  new  model  of  acquisition  governance  ever  more  obvious.  In  order  to  make 
better  use  of  its  defense  dollars  while  contracting  to  acquire  more  advanced  defense 
capabilities,  the  defense  acquisition  community  in  the  United  States  needs  to  improve 
governance  processes  for  complex  acquisitions. 

Over  the  last  decade,  researchers  at  the  Center  for  Strategic  and  International  Studies 
(CSIS)  have  studied  and  analyzed  the  increased  challenges  of  complexity  in  defense 
acquisitions.  Through  workshops,  conferences,  and  publications,  CSIS  has  examined  how 
the  government  purchases  complex  systems,  what  challenges  it  encounters  in  doing  so,  and 
what  key  attributes  of  governance  can  be  modulated  to  provide  more  effective  acquisition. 

The  study  underlying  this  report  explored  one  simple  question:  In  complex  systems  of 
systems,  what  best  practices  contribute  to  better,  more  efficient  acquisitions?  To  answer  this 
question,  the  CSIS  project  team  reviewed  its  past  work  on  complexity  in  acquisitions, 
analyzed  new  scholarship  on  the  subject,  and  conducted  detailed  interviews  with  executives 
from  across  the  defense  acquisition  community.  This  12-month  effort  culminated  with  the 
production  of  a  new  model  of  acquisition  governance  model  presented  in  this  abridged 
version  of  a  longer  technical  report  on  the  research. 

The  governance  framework  starts  with  eight  attributes  of  program  management,  which  CSIS 
developed  under  its  past  research  on  the  subject.  The  study  applied  this  framework  to  seven 
acquisition  case  examples  of  various  sizes  and  complexity  which  have  shown  differing 
degrees  of  program  success.  CSIS  also  conducted  1 1  interviews  with  17  acquisition 
executives  and  stakeholders  covering  each  of  the  case  examples  to  help  identify  success 
drivers  and  good  governance  themes.  Finally,  CSIS  used  the  case  studies  and  interviews  to 
develop  a  model  of  acquisition  governance  that  could  help  strengthen  the  ability  of  acquisition 
leaders  to  more  effectively  and  efficiently  acquire  complex  systems  of  systems  (SoS). 

The  research  illustrates  how  best  practices  in  the  three  most  critical  attributes — level  of 
organizational  focus,  decision-making  authority,  and  enforcement — can  make  the  difference 
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between  success  and  failure  of  an  acquisition  effort.  Best  practices  in  the  remaining  attributes 
also  enable  the  efficient  production  and  procurement  of  effective  systems  of  systems  and 
assist  mission  success  in  large,  complex  acquisitions. 

Readers  should  refer  to  the  full  report  for  supporting  evidence  in  the  form  of  case  studies  and 
interview  summaries. 

Introduction 

Complexity:  The  Problem  Defined 

As  the  defense  needs  of  the  nation’s  warfighters  have  evolved,  acquisition 
preferences  within  the  Department  of  Defense  (DoD)  have  also  transformed  in  a  way  that 
pushes  human  capital,  technical  knowledge,  and  production  assets  away  from  the 
government  and  toward  industry.  However,  existing  models  seem  to  fall  short  of  providing 
an  effective  governance  approach  when  faced  with  the  evolving  challenges  of  complexity. 

Within  complexity  theory,  complex  systems  are  defined  as  those  systems  in  which 
multiple  components  interact  with  and  exert  influence  on  one  another  and  the  various 
factors  in  their  external  environments  over  time.  This  research  focuses  on  complexity  within 
SoS  acquisition  specifically.  As  a  result,  this  paper  uses  complexity  to  describe  systems 
consisting  of  multiple  sub-systems  and  components  that  are  typically  developed  and 
managed  by  more  than  one  organizational  element.  The  existing  literature  has  applied  the 
principles  of  complexity  theory  extensively  to  such  fields  as  economics  and  business 
management.  This  research  aims  to  contribute  to  the  literature  by  identifying  best  practices 
specific  to  the  challenges  defense  and  federal-civil  acquisition  managers  confront  in  their 
SoS  acquisition  efforts.  This  focus  is  narrow,  but  as  with  other  work  that  has  been 
undertaken  on  complexity,  its  implications  are  applicable  to  other  disciplines. 

Complex  systems  present  four  related  problems  for  defense  acquisition  planners  and 
decision-makers.  First,  the  conceptual  size  of  a  complex  system  is  larger  than  a  traditional 
system.  For  example,  ground  vehicles  used  in  the  early-  to  mid-20th  century  were  treated  as 
one  product;  their  interactions  with  other  materiel  were  not  negligible,  but  those  interactions 
could  be  managed  on  a  tactical  level  post-deployment.  Conversely,  today’s  ground  vehicles 
contain  their  own  sub-systems.  A  personnel  carrier  might  have  dozens  of  its  own  active  and 
passive  protection  systems  interacting  with  the  vehicle  and  the  external  environment,  a 
number  of  different  communications  devices,  and  various  on-board  weapons  platforms,  to 
name  a  few.  Each  of  these  systems  has  its  own  acquisition  demands  above  and  beyond  the 
base  requirements  of  the  vehicle  itself. 

Second,  each  of  the  sub-systems  within  a  system,  as  well  as  the  systems  external  to 
it,  interacts  with  others.  Some  of  those  interactions  can  be  forecast  to  some  degree,  but 
forecasts  will  have  limited  accuracy.  Furthermore,  interactions  can  produce  cascading 
interactions  and  change  the  behavior  of  other  sub-systems  or  the  system  as  a  whole.  This 
contributes  to  a  third  problem  presented  by  complexity,  namely  that  the  output  of  a  system  is 
not  equal  to  the  sum  of  its  inputs.  The  interactions  can  create  amplified  or  diminished 
outcomes.  This  is  a  driver  of  complexity  in  defense  platforms,  as  the  multiplying  effects  of 
complementary  systems  can  create  new  and  more  advanced  defense  technologies. 
However,  as  with  the  interactions  themselves,  it  is  difficult  to  forecast  the  individual  and 
aggregate  results  of  the  system  and  sub-system  interactions,  much  less  to  harness  them  for 
a  desired  outcome. 

Finally,  complexity  is  a  management  problem  because  it  operates  outside  the 
boundaries  of  traditional  linear  approaches  to  acquisition  governance.  This  is  central  to  the 
objective  of  this  report.  Traditional,  risk-averse  governance  approaches  to  designing, 
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engineering,  developing,  and  procuring  defense  weapons  and  platforms  are  not  adequately 
flexible  to  manage  the  unpredictable  interactions  of  systems,  sub-systems,  and  the  external 
environment.  They  are  based  on  static  program  objectives  to  acquire  dynamic  technologies. 
Those  technologies  then  often  evolve  at  widely  different  rates  of  development,  and 
upgrading  platforms  with  long  life  cycles  can  be  difficult. 

Models  of  SoS  Governance 

In  order  to  understand  the  problem  presented  by  complexity,  it  is  important  to  have  a 
framework  with  which  to  conceptualize  complex  acquisition.  Broadly  speaking,  there  are  two 
ways  to  conceptualize  efforts  to  acquire  systems  of  systems:  the  traditional  model  and  the 
enterprise  model. 

In  the  traditional  model  of  acquisition  governance,  a  program  office  or  a  similar 
central  authority  mandates  and  governs  the  capabilities  comprising  a  systems  of  systems. 
Linear  acquisition  activities  follow  defined  user  needs,  typically  in  the  form  of  broad  tactical 
requirements.  The  acquisition  activities  begin  with  overarching  technical  requirements, 
which  inform  offers  or  bids  from  government  and  industry  suppliers.  Program  offices  analyze 
bids  for  their  closeness-of-fit  to  certain  criteria  and  select  a  winning  solution.  The  winning 
contractor  then  develops,  produces,  and  deploys  the  systems  of  systems  (see  Figure  1). 


Figure  1.  Traditional  Acquisition  Governance 

Note.  This  figure  comes  from  CSIS  analysis.  “Acquisition  Activities”  was  adapted  from 
standard  defense  acquisition  protocols,  as  outlined  in  DoD  guidance  documents  (2008,  p. 

12). 
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Challenges  pursuant  to  this  model  emerge  from  its  linearity.  The  case  studies  identify 
three  interrelated  ways  in  which  the  linearity  of  the  traditional  model  challenges  the  ability  of 
the  DoD  to  execute  complex  SoS  efforts  and  to  deliver  capabilities  on-cost  and  on-schedule. 
First,  individual  stages  of  the  acquisition  process  have  difficulty  responding  to  changes  in 
user  needs  and  the  external  environment  once  these  inputs  have  set  the  technical 
requirements.  Second,  efforts  to  preempt  changes  in  those  inputs  often  result  in  acquisitions 
of  immature  technologies  with  high  levels  of  risk.  Finally,  as  delays  occur  in  the  development 
of  components  of  the  systems,  parent  systems  cannot  adapt,  which  in  turn  can  create 
setbacks.  This  effect  can  occur  in  any  complex  effort  regardless  of  the  model,  but  the 
structure  of  the  traditional  model  of  governance  exaggerates  this  effect. 

In  contrast  with  the  traditional  model,  the  enterprise  model  of  acquisition  governance 
breaks  SoS  acquisition  into  different  layers  of  authority.  This  characteristic  is  a  key 
difference  between  the  enterprise  model  and  the  traditional  model,  which  employs  a  top- 
down  unified  approach  to  governance.  The  division  of  authority  allows  different  stages  of  an 
SoS  acquisition  effort  to  influence  one  another  in  separate  bilateral  relationships.  It  also 
enables  individual  stages  of  acquisition  to  adapt  to  changes  in  the  external  environment. 
These  characteristics  comprise  a  more  agile  approach  to  acquisition  (see  Figure  2). 
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Figure  2.  Enterprise  Acquisition  Governance 

Note.  This  figure  comes  from  CSIS  Analysis. 

As  illustrated  by  Figure  1  in  the  previous  section,  the  traditional  model  treats  the 
external  environment  as  its  own  unit.  This  allows  environmental  factors  to  inform  user  needs 
but  makes  it  more  difficult  for  them  to  influence  other  stages  of  the  acquisition  effort.  For 
example,  resource  factors  are  critical  to  solutions  analysis  and  selection;  however,  aside 
from  the  influence  of  resource  factors  in  the  initial  stages  of  solutions  analysis,  top-down 
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governance  in  the  traditional  model  complicates  real-time  adjustment  to  changes  in  those 
factors. 


Analysis  of  these  models  indicates  that  layered  authority  is  the  key  differentiator 
between  the  enterprise  and  traditional  approaches  to  acquisition.  However,  CSIS  case  study 
analysis  suggests  that  the  acquisition  community  has  been  selective  about  how  it  divides 
authority,  even  in  those  efforts  most  representative  of  the  enterprise  model.  This  research 
uses  the  eight-attribute  governance  framework  to  observe  how  individual  attributes  relate  to 
each  case’s  ability  to  manage  complexity  internally  and  externally.  The  cases  show  that  the 
ability  to  manage  complexity  directly  correlates  with  the  degree  to  which  the  responsible 
stakeholders  have  independent  authority  in  each  attribute  for  which  they  are  accountable. 

The  seven  cases  included  in  this  study  include  two  cases  with  more  traditional 
governance  models  and  five  with  governance  models  that  were  closer  to  the  enterprise 
model.  Here  are  the  seven  cases. 

Traditional  Governance  Cases: 

1 .  Future  Combat  System  (FCS) 

2.  Integrated  Deepwater  System  (Deepwater) 

Enterprise-Wide  Governance  Cases: 

3.  Counter  Rocket,  Artillery,  and  Mortar  (C-RAM) 

4.  Distributed  Common  Ground  System  (DCGS) 

5.  Global  Nuclear  Detection  System  (GNDS) 

6.  Harvest  Hawk 

7.  Maritime  Domain  Awareness  (MDA) 

Attributes  of  SoS  Governance 

CSIS  developed  its  framework  for  analysis  of  governance  in  complex  acquisitions  to 
help  understand  the  drivers  of  success  in  SoS  acquisition.  The  framework  is  the  product  of 
research  on  SoS  governance  models,  interviews  with  program  stakeholders  and  industry 
leaders,  and  a  findings  refinement  process  involving  input  from  SoS  experts.  It  consists  of 
eight  attributes  that  collectively  represent  concerns,  questions,  and  issues  that  must  be 
addressed  for  an  organization  to  succeed  in  acquiring  a  complex  systems  of  systems.  The 
importance  and  significance  of  these  attributes  varies  depending  on  the  system  to  which  the 
framework  is  being  applied. 

The  eight  governance  attributes  are  as  follows: 

•  Level  of  organizational  focus:  the  level  at  which  SoS  governance  occurs 
within  the  organization.  This  is  not  the  same  as  systems/capabilities  focus  or 
technical  focus,  both  of  which  are  outside  the  scope  of  the  CSIS  SoS 
governance  analysis. 

•  Integration  of  functional  end-user  needs:  the  mechanisms  and  frequency  with 
which  the  functional  needs  of  end-users  are  built  into  the  systems  of  systems 
and  at  which  points  in  the  process  this  incorporation  occurs. 

•  Decision-making  authority:  the  governance  mechanisms  for  SoS  delivery, 
including  how  budget  is  allocated,  standards  set,  tradeoffs  managed,  and 
inconsistencies  adjudicated. 

•  Enforcement:  the  mechanisms  and  level  of  oversight  by  which  the  objectives 
of  the  SoS  capability  to  be  delivered  are  ensured. 
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•  Workforce:  the  examination  of  SoS  workforce  structures,  unity  of  mission, 
and  capability  development  through  use  of  contracting. 

•  Incentive  structure:  the  alignment  between  the  enterprise’s  goals  and  the 
incentive  and  reward  structures  of  the  stakeholders  and  organizations  that 
implement  them. 

•  Knowledge  ownership/access  to  knowledge:  the  accessibility  of  information 
regarding  the  operating  environment,  technical  standards,  and  the  other  parts 
of  the  systems  of  systems. 

•  Risk  assessment/risk  management:  risk  assessments  and  management 
strategies  tailored  to  the  mission  accomplishment  and  the  flexibility  and 
resilience  required  for  delivering  systems  of  systems  in  the  face  of 
unforeseen  developments. 

This  research  examined  seven  case  studies  to  profile  their  characteristics  in  each  of 
these  attribute  areas.  It  then  conducted  interviews  with  acquisition  executives  and  other 
program  stakeholders  to  understand  which  attributes  were  most  important  in  producing  the 
successes  of  each  effort,  as  well  as  those  areas  in  which  each  effort  did  not  perform 
optimally.  A  detailed  analysis  of  each  case  study  is  available  in  the  full  report  associated 
with  this  proceedings  paper. 

Results  of  CSIS  Interviews  With  SoS  Program  Stakeholders 

Case  studies  on  seven  example  SoS  acquisition  efforts  comprised  the  core  of  this 
CSIS  research  effort.  These  case  studies  facilitated  analysis  of  the  specific  governance 
attributes  for  each  case  and  provided  important  qualitative  gauges  on  their  importance.  The 
results  of  these  interviews  suggest  that  an  agile  and  multi-layered  governance  process  can 
lessen  the  operational  burden  of  managing  and  delivering  complex  systems  of  systems. 
Furthermore,  strong  enforcement  mechanisms  are  necessary  to  ensure  outcomes  are  in  line 
with  expectations. 

The  CSIS  research  team  conducted  11  interviews  in  total  with  the  participation  of  17 
acquisition  executives.  Each  interview  lasted  about  1-2  hours  and  was  semi-structured  to 
allow  interviewees  to  stray  from  the  explicit  governance  attributes  and  provide  more  general 
commentary  on  acquisition  challenges. 

The  interviews  suggested  that  five  themes  are  critical  to  understanding  governance 
of  complex  acquisition.  These  themes  are  presented  here  in  order  of  importance  to 
governance. 

Theme  #1 :  Stakeholders  must  maintain  focus  on  the  end-result  of  an 
acquisition  effort  at  the  enterprise  level. 

First,  complex  acquisition  governance  requires  that  all  stakeholders  maintain  focus 
on  the  end-result  at  the  enterprise  level.  This  structural  issue  is  central  to  a  system’s  ability 
to  operate  in  an  environment  consisting  of  multiple  other  systems.  Interviewees  suggest  that 
a  narrow  focus — focus  on  a  specific  product  or  capability  set,  for  example — compromises 
the  value  of  a  single  system’s  acquisition  to  the  overall  objectives  of  the  systems  of  systems 
and  its  user  communities.  This  narrow  focus  also  contributes  to  demand  uncertainty  in 
instances  where  stakeholders  simultaneously  develop  multiple  competing  capabilities  to 
accomplish  the  same  end  goal. 
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One  case  that  CSIS  examined,  Future  Combat  Systems  (FCS),  offers  several 
insights  about  the  impact  of  a  product-  or  program-level  focus  on  the  success  of  an 
acquisition  effort.  A  former  DoD  acquisition  official  involved  in  overseeing  FCS  said  that 
many  in  the  acquisition  community  charged  with  approving  the  program  at  its  various 
decision  milestones  were  concerned  that  FCS  network  components  seemed  to  duplicate 
functions  available  with  existing  technologies.  Furthermore,  the  source  continues,  FCS  had 
no  roadmap  for  integration  with  interacting  systems,  particularly  the  Warfighter  Information 
Network-Tactical  (WIN-T),  an  all-encompassing  Army  tactical  network  program,  and  Joint 
Network  Node  (JNN),  a  SatCom  system.  One  interviewee  criticized  the  program  for 
attempting  to  build  its  own  operating  systems  from  scratch  rather  than  use  existing  operating 
systems  able  to  interface  with  WIN-T  and  JNN.  The  engineers  had  no  plan  for  bringing 
those  pieces  together,  and  testing  their  interactions  required  that  the  entire  system  first  be 
fielded,  the  interviewee  reports. 

Interviews  with  several  program  executives  and  stakeholders  emphasized  the 
importance  of  enterprise-level  focus.  In  cases  such  as  FCS  where  focus  is  narrow,  problems 
emerge  where  the  feasibility  of  end-state  integration  is  unclear.  Integration  problems  also 
surface  in  cases  where  enterprise-focus  is  present,  however.  For  example,  in  another  CSIS 
case  study,  Maritime  Domain  Awareness  (MDA),  acquisition  executives  maintained  a  focus 
on  the  objectives  of  the  entire  enterprise  but  were  presented  with  challenges  to  enterprise¬ 
wide  integration  all  the  same. 

Theme  #2:  A  layered  decision-making  structure  with  distinct,  delegated 
authority  must  be  clearly  established  at  the  outset  of  complex  acquisition  efforts. 

The  reason  for  integration  problems,  even  in  those  programs  in  which  organizational 
focus  is  held  at  the  enterprise  level,  underscores  a  second  theme  about  complex  acquisition 
governance.  It  suggests  that  a  layered  decision-making  structure  with  distinct,  delegated 
authority  must  be  clearly  established  at  the  outset  of  complex  acquisition  efforts  in  order  for 
their  level  of  organizational  focus  to  matter. 

Interview  results  show  that  the  various  process  constraints  faced  by  each  of  the 
acquisition  case  studies — for  example,  clarity  and  consistency  of  scope,  availability  of 
personnel,  inventory  of  resources,  and  so  forth — are  due  in  part  to  their  decision-making 
structures.  Process  constraints  also  translate  to  poor  outcomes  in  cases  where  decision¬ 
making  structures  were  not  designed  to  meet  the  specific  needs  of  an  acquisition  effort,  or 
where  decision-making  authority  was  weak  or  non-existent. 

The  MDA  effort  demonstrates  the  challenges  created  by  weak  and  heavily 
centralized  decision-making  authority.  In  that  case,  interviewees  report  that  the  Executive 
Agent  for  MDA  (EAMDA)  has  notional  leadership  over  stakeholder  agencies  but  no  real 
decision-making  authority.  Instead,  it  serves  in  the  role  of  what  one  interviewee  calls  a 
“broker  and  a  cheerleader.”  Formally,  EAMDA  issued  an  Initial  Capabilities  Document  (ICD) 
to  allow  material  solutions  development  under  the  standard  defense  acquisition  process. 
However,  there  is  no  formal  lead  for  requirements  generation  or  subsequent  acquisition 
decisions.  As  a  result,  stakeholder  agencies  have  different  perceptions  of  the  MDA  charter 
and  what  technologies  satisfy  its  core  mission  objectives.  These  definitional  issues  emerge 
in  the  absence  of  strong  central  leadership.  Interviewees  support  this  observation  and  report 
similar  challenges  in  the  Global  Nuclear  Detectiong  System  (GNDS)  case  study,  which  is 
also  examined  in  the  full  technical  report. 
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Figure  3.  Distributed  Common  Ground  System  Conceptual  Governance  Structure 

and  Key  Acquisition  Processes 

Note.  This  figure  comes  from  CSIS  analysis. 

CSIS  analysis  of  the  Distributed  Common  Ground  System  (DCGS)  contrasted  with 
the  MDA  case  in  important  ways.  The  DCGS  decision-making  structure  is  relatively  well- 
suited  to  the  mission  objectives  of  that  acquisition,  particularly  in  its  decentralization.  One 
DCGS  executive  reports  that  DCGS  decision-making  is  conceptually  separated  into 
recommendation  and  implementation  functions.  Figure  3  illustrates  this  concept  in  the 
DCGS  organizational  structure  and  the  key  acquisition  processes. 

The  DCGS  Steering  Board  serves  as  a  strong,  central  leadership  entity  to  provide 
capabilities  guidance  and  keep  development  and  procurement  efforts  aligned  with  core 
mission  requirements.  The  Army,  Air  Force,  Navy,  and  Marine  Corps  each  have  their  own 
lead  entities  for  the  implementation  functions.  Each  of  the  service  components  refers  to  the 
ICD  to  draft  Capabilities  Production  Documents  (CPDs)  for  each  of  their  respective  systems. 
The  DCGS  Steering  Board  then  reviews  the  new  capabilities  to  determine  their  fit  with  the 
standards  and  specifications  established  in  the  ICD  and  governed  by  a  central 
interoperability  framework,  the  Defense  Intelligence  Information  Enterprise  (DI2E).  Similar  to 
MDA  and  GNDS,  this  workflow  represents  an  agile  approach  to  decision-making  that 
empowers  end-user  communities  with  substantially  different  requirements  to  make  their  own 
procurement  decisions.  However,  it  is  supplemented  by  strong,  formalized  central 
leadership. 
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Theme  #3:  Decision-makers  must  design  effective  management  and 
enforcement  mechanisms  and  integrate  them  into  the  acquisition  process. 

Interviewees  suggest  that  enforcement  mechanisms  are  the  critical  component  tying 
together  the  enterprise  focus  with  product-level  decision-making.  In  order  to  maintain  short 
chains  of  authority  between  the  technical  communities  in  the  production  chain  with  the 
enterprise-level  planners  at  the  top,  decision-makers  must  design  effective  management 
and  enforcement  mechanisms  and  integrate  them  into  the  acquisition  process.  Formalized 
program  management  mechanisms  consisting  of  a  well-defined  set  of  performance  criteria 
have  the  advantage  of  lessening  the  process  burden  of  complex  acquisition.  With 
complementary  incentives  and  metrics  for  monitoring  performance,  enhanced  program 
management  can  also  contribute  to  improved  cost  and  schedule  outcomes. 

The  process  advantages  of  effective  program  management  are  apparent  in  the 
DCGS  case.  The  DCGS  effort  runs  an  overarching  recommendations  and  review  process 
through  the  DCGS  Steering  Board,  but  delegates  acquisition  implementation  to  individual 
end-user  communities  based  on  a  common  set  of  standards  established  in  the  DI2E 
Framework.  At  the  systems  level,  the  acquiring  service  can  approach  risk  in  technology 
acquisition  on  a  case-by-case  basis.  The  acquiring  armed  forces  component  can  also 
design  its  contract  or  contracts  based  on  the  scale  of  a  particular  acquisition  item,  or 
whether  the  acquisition  is  developmental  or  non-developmental.  At  the  enterprise  level,  the 
DCGS  Steering  Board  has  a  mediation  process  through  which  it  can  deny  efforts  that  it 
determines  to  be  too  risky  or  that  do  not  contribute  to  the  DCGS  mission  objectives. 

The  effectiveness  of  the  DCGS  program  management  structure  is  particularly 
apparent  when  compared  with  the  management  systems  used  in  other  large  complex 
acquisition  cases,  such  as  FCS  and  Integrated  Deepwater  System  (Deepwater).  In  these 
cases,  acquisition  was  directed  through  single,  service-centric  programs  of  record.  An 
overarching  rule-set  was  applied  to  all  subordinate  efforts.  Sources  interviewed  suggest 
DCGS  has  avoided  the  process  rigidity  that  plagued  FCS  and  Deepwater  in  part  because  of 
its  separation  of  top-down  program  recommendations  and  bottom-up  program 
implementation. 

Theme  #4:  Program  leadership  must  prevent  vested  interests  and  cost 
concerns  from  becoming  barriers  to  knowledge  ownership. 

Organizational  focus  at  the  enterprise  level,  driven  by  clearly  delegated  decision¬ 
making  authority  and  supported  by  effective  enforcement  mechanisms,  creates  the 
necessary  underpinnings  of  a  complex  governance  structure.  However,  any  one  of  these 
structural  elements  can  be  compromised  when  stakeholders  are  unable  or  unwilling  to  share 
information.  This  points  at  a  fourth  theme  in  complex  acquisition  governance:  In  order  to 
execute  a  complex  acquisition  effort  effectively  and  efficiently,  agencies  and  offices 
partnered  in  the  technical  development  of  systems  as  well  as  the  duties  of  program 
management  must  be  able  to  access  and  share  information  among  themselves.  Oftentimes, 
barriers  to  knowledge  access  can  inhibit  the  success  of  a  complex  project.  Conversations 
with  interviewees  suggest  that  knowledge  access  also  depends  on  strong  incentives  for 
active  sharing  of  information  about  technologies,  program  goals,  and  progress  toward 
established  milestones. 

Interviewees  indicate  that  there  are  two  types  of  barriers  to  effective  information 
sharing  and  knowledge  access.  The  first  barrier  is  knowledge  protectionism.  Especially  in 
instances  of  collaboration  between  private  sector  stakeholders,  conversations  with 
interviewees  suggest  that  companies  are  more  likely  to  withhold  information  from  partner 
entities  due  to  competition  for  other  government  contracts.  This  concern  was  especially 
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strong  in  early  efforts  to  make  the  DCGS  Integration  Backbone  (DIB)  software  package 
open-source.  In  an  effort  to  facilitate  software  revisions  to  help  meet  the  needs  of  individual 
end-user  communities,  the  DCGS  Steering  Board  and  DI2E  Integration  Council  were 
strongly  supportive  of  revealing  the  DIB  source  code  to  the  development  community. 
However,  according  to  one  program  source,  the  DIB  parent  developer  resisted  efforts  to 
make  the  software  open-source.  In  the  end,  DCGS  customer  offices  were  able  to  compel 
the  contractor  to  share  its  knowledge. 

Program  challenges  resulted  from  early  resistance  to  revealing  the  DIB  software 
code.  By  compelling  its  primary  software  contractor  to  open  its  code  for  other  users  to 
access  and  develop,  the  DCGS  program  office  demonstrated  how  a  large  organization  can 
work  to  reduce  the  impact  of  knowledge  protectionism.  Using  the  code,  DIB  challenger 
MarkLogic  successfully  developed  a  version  of  the  DIB  based  on  Extensible  Markup 
Language  (XML,  an  open-standards  software  encoding  format)  to  help  meet  the  needs  of  a 
sub-segment  of  the  broader  end-user  community,  the  U.  S.  Special  Operations  Command 
(USSOCOM),  which  had  a  need  for  a  high-performance  reconnected-ops  DI2E. 

A  second  barrier  to  information  sharing  is  created  by  differing  levels  of  information 
management  capabilities.  Levels  of  technical  astuteness  can  be  widely  different  between 
government  customers  and  industry  suppliers.  This  problem  is  compounded  by  varied 
clearance  privileges  required  for  access  to,  and  use  of,  compartmentalized  information. 
Sources  interviewed  indicate  this  problem  has  complicated  several  systems-of-systems 
acquisitions.  Interviewees  suggest  that  information  needs  among  stakeholders  in  MDA  are 
challenged  both  by  the  breadth  of  information  collected  and  the  need  to  filter  it  to  lower 
levels  of  clearance,  especially  unclassified  civil  users  in  the  Coast  Guard.  For  participants  in 
the  Hercules  Airborne  Weapons  Kit  effort — known  more  commonly  as  Harvest  HAWK — data 
relevant  to  that  program  were  only  accessible  through  a  single  computer  network,  requiring 
physical  ownership  of  the  data  for  effective  information  sharing.  Sources  point  to  these 
barriers  as  real  and  addressable  challenges  to  acquiring  and  integrating  systems  of 
systems. 

Theme  #5:  Leaders  at  each  level  of  the  systems  of  systems  must  be  adaptable 
to  changes  that  result  from  human  behavior. 

A  fifth  and  final  theme  of  complex  governance  reported  by  interviewees  observes 
that  leaders  at  each  level  of  the  systems  of  systems  must  be  adaptable  to  changes  resulting 
from  human  behavior.  This  theme  is  applicable  not  just  to  the  acquisition  of  complex 
systems  of  systems  involving  many  components  and  stakeholders  but  also  to  other  systems 
in  the  natural  world.  Human  behavior  is  the  ultimate  uncertainty,  and  agility  is  critical  to 
absorbing  the  impact  of  its  changes. 

Within  SoS  acquisition  efforts  specifically,  effective  governance  will  recognize 
changes  in  the  needs  of  a  system’s  end-users  early  and,  based  on  that  recognition, 
reevaluate  the  approaches  to  a  planned  acquisition.  This  feedback  loop  can  be  formalized 
through  component-level  proposals  for  new  capabilities  (MDA).  It  can  also  be  more  ad  hoc, 
incorporating  standards-based  innovations  developed  by  end-users  into  the  completed 
systems  of  systems  (DCGS).  In  either  case,  the  traditional  approach  to  acquisition  appears 
to  limit  responsiveness  to  end-user  needs. 

In  general,  effectiveness  appears  to  share  a  strong  correlation  with  cases  that  were 
initiated  with  a  clear  end-user  in  mind.  For  example,  the  Harvest  HAWK  acquisition  was  born 
out  of  the  necessity  to  provide  air  support  for  Marines  in  Afghanistan.  Marines  deployed  to 
train  Afghan  security  forces  developed  an  urgent  need  for  air  support  when  their  training  unit 
encountered  persistent  attacks  from  adversary  forces.  Information  obtained  from  the 
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Marines  provided  an  impetus  for  the  Harvest  HAWK  acquisition  effort,  making  this  case 
particularly  representative  of  the  effective  use  of  human  information  inputs  to  produce  and 
procure  capabilities  specifically  answering  the  needs  of  the  end-user. 

An  additional  aspect  of  human  input  is  the  workforce  that  a  systems-of-systems  effort 
has  at  its  disposal.  A  good  technical  and  program  management  workforce  can  make  the 
difference  between  an  effort’s  success  and  failure.  Recruiting  a  truly  great  workforce  can 
dramatically  improve  both  the  magnitude  of  a  system’s  success  as  well  as  the  process 
required  to  develop  and  procure  it.  Harvest  HAWK  provides  an  example  of  the  importance  of 
skilled  technical  personnel  supplemented  by  experienced  program  managers.  One  source 
indicated  that  Harvest  HAWK’s  success  in  delivering  a  complex  capability  in  a  small  period 
of  time  is  attributable  in  part  to  the  ability  of  the  program  management  to  take  advantage  of 
the  technical  workforce  available  through  Naval  Air  Systems  Command  (NAVAIR). 

Several  other  cases  illustrate  that  a  strong  program  management  workforce  is  not  as 
effective  in  delivering  a  system-of-systems  capability  when  it  has  a  weak  technical  workforce 
at  its  disposal.  Sources  suggested  that  workforce  strength  is  not  measured  by  technical 
skillsets  alone.  Rather,  complex  acquisition  efforts  benefit  when  their  workforce  is 
empowered  to  use  its  technical  know-how  to  innovate  new  approaches  to  solving  problems. 

Although  technical  astuteness  tends  to  reside  with  human  capital  in  engineering 
functions,  program  managers  often  mistake  general  capabilities  needs  with  prescriptive 
technical  requirements  under  the  existing  framework  for  acquisition.  Sources  interviewed 
suggest  that  there  is  structural  resistance  to  bottom-up  innovation  of  new  approaches  to 
meeting  capabilities  demands,  but  several  cases  included  examples  where  changing  both 
personnel  and  their  empowerment  led  to  breakthroughs  on  problems. 

Analysis  of  Best  Practices 

CSIS  independently  assessed  success  factors  in  the  attributes  and  supplemented 
analysis  with  primary  input  from  sources  interviewed.  This  analysis  extracted  the  best 
practices  presented  here.  This  approach  facilitated  analysis  by  incorporating  the  existing 
literature  on  complex  systems  with  the  expertise  of  acquisition  executives  involved  in 
individual  or  multiple  cases. 

Critical  Best  Practices 

An  assessment  of  the  case  studies  shows  that  success  in  three  attributes  in 
particular  is  critical  to  SoS  acquisition.  Those  attributes  are  level  of  organizational  focus, 
decision-making  authority,  and  enforcement.  The  attributes  are  closely  related,  and  strong 
performance  in  any  one  is  dependent  upon  performance  in  the  other  two. 

Organizational  focus  at  the  enterprise  level  is  critical  to  enabling  SoS  integration  and 
facilitating  flexibility  for  quick  and  substantive  response  to  changes  in  the  external 
environment.  Although  critical,  enterprise-level  focus  is  limited  when  it  is  not  supplemented 
with  some  program-level  focus  on  individual  capabilities.  The  DCGS  case,  for  example, 
shows  the  value  of  allowing  programs  and  sub-system  acquisition  efforts  to  govern 
themselves  to  a  certain  point. 

The  key  to  differentiating  these  two  levels  of  focus  and  ensuring  that  they  interact 
with  and  respond  to  one  another  is  in  part  related  to  the  decision-making  authority  attribute. 
At  the  program  level,  stakeholders  should  be  delegated  the  authority  to  make  decisions 
about  systems  and  technologies  with  a  low  burden  of  approval  from  the  enterprise  level. 

This  is  especially  important  in  complex  systems  of  systems.  The  organizational  legwork  a 
program  has  to  perform  in  order  to  approve  a  new  sub-system  that  has  newly  developed  or 
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emerged  as  an  end-user  need  can  prevent  the  program  from  timely  and  effective  integration 
of  that  technology  or  capability. 

One  way  to  achieve  the  right  level  of  agility  in  decision-making  delegation  is  to 
establish  open-source  standards  for  new  technologies  and  systems  as  a  replacement  for 
traditional,  formal  oversight  mechanisms.  In  interviews  with  CSIS,  several  Coast  Guard 
executives  indicated  that  it  is  already  implementing  this  best  practice  post-Deepwater  to 
ensure  interoperability  among  C4ISR  systems  across  its  fleet  of  surface  vessels  and  aircraft. 
Interviews  also  indicate  DCGS  revealed  its  DCGS  Integration  Backbone  (DIB)  for  open 
source  use,  allowing  industry  to  create  a  new  and  specialized  XML-based  search  engine  in 
order  to  meet  a  very  specific  end-user  need.  Therefore,  this  best  practice  assists  decision¬ 
makers  at  the  enterprise  level  by  lessening  their  programmatic  requirements,  and  empowers 
decision-makers  at  the  program  level,  as  well  as  end-users  themselves,  to  reflect  SoS 
ecosystem  changes  in  their  planned  acquisition  activities. 

The  enforcement  attribute  links  decision-making  authority  with  level  of  organizational 
focus.  Effective  enforcement  mechanisms  enable  enterprise  governance  authorities  to  keep 
program  managers  on-cost  and  on-schedule  while  easing  the  operational  burdens 
associated  with  delivering  large,  complex  systems  of  systems.  Although  statutory 
mechanisms  for  enforcement  are  established  in  the  DoD’s  standard  acquisition  process, 
case  study  analysis  finds  those  mechanisms  custom-tailored  to  the  specific  demands  of  the 
individual  systems  of  systems  to  be  most  successful.  For  example,  the  DCGS  effort  created 
a  central  authority  to  review  petitions  for  capabilities  from  an  enterprise  perspective,  and 
reserves  the  right  to  deny  a  sub-system  acquisition  effort  in  the  event  that  it  is  determined  to 
be  incompatible  with  the  SoS  objectives.  This  exists  outside  of  the  standard  DoD  acquisition 
enforcement  mechanisms.  Furthermore,  the  case  of  FCS  illustrates  how  existing 
mechanisms  can  in  fact  be  counterproductive  and  conceal  underlying  problems  in  a 
complex  SoS  acquisition  effort. 

Enabling  Best  Practices 

In  addition  to  the  critical  best  practices,  performance  in  the  remaining  attributes 
enables  SoS  acquisition  efforts  to  reach  end  capabilities  with  fewer  cost  and  schedule 
problems.  These  enabling  best  practices  also  contribute  to  greater  alignment  between  a 
systems  of  systems  and  the  needs  of  its  end-users.  Finally,  best  practices  in  the  remaining 
attributes  help  to  mitigate  problems  of  technology  obsolescence  and  help  ease  the  tendency 
for  innovation  in  modern  defense  platforms  to  lag  behind  commercial  industry. 

Successful  SoS  acquisition  efforts  accomplish  greater  integration  of  functional  end- 
user  needs  in  part  through  the  encouragement  of  innovation  at  the  sub-systems  level.  One 
best  practice  in  this  regard  involves  the  establishment  and  publication  of  open  standards  for 
software  and  systems.  The  DCGS  offers  one  example.  In  that  case  study,  DCGS  program 
management  responded  to  a  SOCOM  need  for  high-performance  reconnected  ops  DI2E  by 
pressuring  the  DCGS  prime  contractor  to  release  the  DIB  source  kernel  for  open-source 
collaboration.  Using  the  DIB  package,  MarkLogic,  a  supplier  of  enterprise  database 
software,  created  an  XML  database.  In  the  end,  the  software  proved  valuable  to  SOCOM,  a 
specific  end-user  community  with  unique,  niche  requirements. 

In  addition  to  revealing  systems  for  open-source  innovation,  case  study  analysis  also 
indicates  that  kit-based  and  modular  approaches  to  materiel  development  can  contribute  to 
greater  integration  of  end-user  needs.  The  reverse  of  this  is  the  more  traditional  platform- 
based  development.  Harvest  HAWK  provides  one  example  of  this  best  practice.  In  that 
program,  users  found  the  original  kit  to  be  inconvenient  because  of  its  weapons  release 
mechanism.  Namely,  in  order  to  deploy  missiles,  the  system  had  to  first  depressurize.  The 
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installation  of  a  derringer  door  into  the  kit  corrected  the  issue,  and  was  made  possible  by  the 
fact  that  the  kits  are  not  permanently  installed  on  the  C-130J  platforms.  The  roll-on/roll-off 
nature  of  the  Harvest  HAWK  capability  allows  for  refinements  not  requiring  significant  SoS- 
level  changes.  In  contrast  with  this  best  practice,  systems  acquired  under  a  traditional  model 
show  a  heavily  diminished  ability  to  integrate  changing  end-user  needs.  Future  Combat 
Systems  (FCS),  for  example,  folded  all  capabilities  into  a  slow-moving  platform  acquisition 
effort  in  a  way  that  precluded  changes  to  sub-systems  based  on  new  end-user  needs. 

In  either  of  the  two  best  practices  in  user  needs  integration  (i.e. ,  open  source 
standards  and  kit-based  development),  performance  in  knowledge  ownership  and  access  to 
knowledge  is  an  important  enabler.  For  this  reason,  case  studies  illustrate  that  open-source 
publication  of  standards  and  basic  software  is  a  best  practice  in  the  information  access 
attribute.  However,  revealing  the  foundation  for  a  systems  of  systems  alone  is  not  enough  to 
enable  stakeholders  to  develop  and  refresh  sub-systems.  This  information  is  only 
meaningful  insofar  as  leadership  in  any  SoS  acquisition  effort  recognizes  that  stakeholders 
across  systems  of  systems  vary  widely  in  their  ability  to  access,  understand,  and  use 
information  that  is  available  to  them. 

Systematic  and  institutional  stovepipes  are  one  obstacle  to  information  ownership. 
Other,  less  tangible  obstacles  include  lower  technical  expertise  among  information 
consumers,  lack  of  sufficient  resources  to  handle  the  financial  and  technical  costs  of 
information,  and  compatibility  issues  in  the  recipients’  systems  with  the  format  or  type  of 
information.  Therefore,  one  best  practice  to  facilitate  knowledge  ownership  and  access  to 
information  is  to  campaign  for — or  in  the  event  that  the  provider  of  information  is  also  the 
lead  stakeholder  for  a  systems  of  systems,  to  approve  and  make  available — the  budgetary 
and  technical  resources  necessary  for  information  management. 

Workforce  best  practices  comprise  a  third  enabling  factor  in  SoS  acquisition. 
Identifying,  recruiting,  and  retaining  the  most  appropriate  personnel  for  any  SoS  acquisition 
effort  can  substantially  impact  the  effort’s  process  and  outcomes.  However,  identifying  best 
practices  in  this  attribute  is  particularly  difficult  for  two  reasons.  First,  human  inputs  are 
unpredictable.  In  the  case  of  Harvest  HAWK,  replacing  a  single  engineer  rescued  the 
program  from  a  months-long  logjam  when  the  new  engineer  identified  an  error  in  a  sub¬ 
system’s  code  that  had  been  inhibiting  the  proper  function  of  the  systems  of  systems. 
Second,  workforce  is  a  complex  system  in  itself;  technical  and  programmatic  personnel 
interact  with  one  another  at  different  levels  with  unpredictable  impacts. 

Analysis  of  the  case  studies  indicates  that  one  best  practice  in  the  workforce  attribute 
is  to  balance  programmatic  and  technical  personnel  based  on  the  specific  needs  of  a 
program.  At  the  enterprise  level,  the  ideal  workforce  is  weighted  toward  programmatic 
workers  with  some  technical  personnel  available  to  provide  overarching  direction.  At  the 
systems  level,  technical  personnel  are  more  prevalent  in  the  ideal  workforce.  Weighting 
toward  technical  personnel  rather  than  programmatic  personnel  allows  the  workforce  to 
avoid  parochial  interests  and  provide  metrics-based  assessments  of  progress  toward  a  sub¬ 
system’s  development. 

Incorporating  recent  and  one-time  end-users  into  the  workforce  is  an  additional  best 
practice  that  enhances  the  alignment  of  outputs  with  the  need  they  are  expected  to  satisfy. 
The  Deepwater  family  of  vessels  has  been  incorporating  this  best  practice  into  its  efforts  to 
acquire  the  vessels  formerly  comprising  that  systems  of  systems.  The  presence  of  decision¬ 
makers  with  recent  field  experience  in  the  Harvest  HAWK  initiative  also  contributed  to  that 
program’s  success  in  producing  an  effective  systems  of  systems  to  meet  an  urgent 
operational  need  on  the  part  of  the  Marine  Corps  for  close  air  support. 
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Regarding  the  incentive  structure  of  an  acquisition  effort,  analysis  indicates  fostering 
and  satisfying  greater  mission  commitment  among  stakeholders  is  an  ideal  motivator. 
Although  mission  commitment  seems  intangible  and  difficult  to  effect,  the  case  studies 
highlight  different  ways  to  enhance  the  commitment  of  personnel  throughout  an  acquisition 
effort’s  hierarchy  to  the  effort’s  values  and  desired  end-state.  For  example,  having  strong 
budgetary  advocates  at  the  executive  level  communicates  to  stakeholders  that  their  work  in 
the  acquisition  of  a  systems  of  systems  is  valuable  to  the  end-user  and  to  the  organization 
as  a  whole.  Analysis  indicates  that  personnel  management  tools  can  provide  additional 
incentives.  For  example,  as  shown  in  the  Deepwater  case,  the  strategic  appointment  of 
accomplished  personnel  to  leadership  positions  can  inspire  confidence  throughout  a 
program.  Similarly,  accountability  at  each  level  of  the  personnel  hierarchy  provides  a 
disincentive  to  poor  performance. 

Case  study  analysis  also  illustrates  that  acquisition  managers  can  influence  an 
effort’s  success  through  incentives  aimed  at  other  systems  and  organizations.  Most  notably, 
high  barriers  to  the  entry  of  new  technologies  into  an  acquisition  effort  act  as  a  disincentive 
to  mid-stream  innovation  and  damage  the  long-term  effectiveness  of  a  systems  of  systems. 
The  Harvest  HAWK  case  offers  several  examples  of  barriers  to  entry  and  their  impact  on  an 
effort’s  process  and  outcomes.  At  the  front  end,  the  Marine  Corps  did  not  conduct  a 
competition  for  source  selection  on  the  Harvest  HAWK  contract.  Challenges  that  emerged 
later  in  the  development  process  may  have  surfaced  sooner  if  competitive  offerings  had 
been  evaluated.  In  contrast  with  this  situation,  lower  barriers  to  entry  of  competitive  and 
more  recent  technologies  in  DCGS  facilitated  the  emergence  of  a  competitive  database 
solution  that  ultimately  improved  the  operability  of  the  entire  systems  of  systems. 

These  best  practices  above  represent  ways  to  structure  a  complex  acquisition  effort 
to  improve  effectiveness  and  efficiency  based  on  seven  of  the  eight  attributes  analyzed. 
Identifying  best  practices  in  risk  assessment,  the  final  attribute,  presents  two  significant 
challenges.  First,  risk  is  an  inherent  and  unavoidable  aspect  of  complexity.  In  addition  to 
uncertainties  originating  in  the  development  and  procurement  of  individual  systems,  the 
interactions  between  and  among  those  systems  also  creates  potential  areas  of  risk.  Second, 
risks  emerge  in  different  stages  of  any  given  project  in  ways  that  cannot  be  predicted, 
complicating  the  universal  application  of  best  practices.  For  example,  the  Army’s  approach 
to  FCS  exhibited  greater  risks  in  the  early  stages  of  system  development.  The  effort  was 
particularly  risky  in  its  requirements  generation  processes,  which  led  to  the  selection  of 
immature  and  untested  technologies.  Conversely,  Harvest  HAWK  was  non-developmental 
by  default,  and  requirements  in  the  early  stages  of  that  acquisition  were  less  risky.  It  would 
appear  that  Harvest  HAWK  has  adopted  this  as  a  best  practice  to  reduce  and  manage  risk. 
In  fact,  risks  were  just  displaced  elsewhere.  They  emerged  when  the  Marine  Corps  began 
piecing  existing  systems  together  and  creating  new  interactions  for  which  the  sub-systems 
had  not  been  designed. 

Difficulties  aside,  analysis  indicates  best  practices  for  the  management  and 
assessment  of  risk  exist,  but  they  should  be  evaluated  for  their  fit  with  the  specific  objectives 
and  challenges  of  an  SoS  acquisition  effort  and  its  unique  sub-systems  and  interactions. 

A  New  Model  of  Systems-of-Systems  Acquisitions 

By  combining  the  results  of  its  case  study  analysis  with  the  themes  gathered  through 
the  primary  research  interviews,  CSIS  developed  the  acquisition  model  presented  in  Table 
1 .  This  model  reflects  the  best  practices  described  earlier  in  this  report. 
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Table  1.  Best  Practices  in  Systems-of-Systems  Acquisition 


Governance  Attributes 

Program  Characteristics 

Level  of  organizational 
focus 

•  Program-level  focus  supplemented  by  overarching  enterprise-level 
governance  and  short  authority  chains 

Decision-making  authority 

•  Stakeholders  are  given  relative  autonomy  to  make  systems-level 
decisions  based  on  standards  and  common  operating  environments 
installed  at  the  enterprise  level 

•  A  central  governance  oversight  body  holds  auditing  and  enforcement 
powers  in  order  to  maintain  commitment  to  systems-of-systems  core 
objectives 

Enforcement 

•  Enterprise-level  authorities  maintain  ability  to  review  and  revise  gate 
decisions  as  sub-systems  evolve  and  needs  change 

•  Clear,  systems-specific  reporting  requirements  are  established  based  on 
technological  maturity  and  projected  development  schedules  to  support 
enforcement  authority 

Integration  of  functional 
end-user  needs 

•  Kit-based,  modular  systems  allow  timely  integration  of  end-user  needs 
and  changes  in  the  external  environment 

•  End-users  are  encouraged  and  empowered  to  develop  their  own  sub¬ 
systems  solutions  in  compliance  with  established  SoS  standards 

Workforce 

•  Systems-level  technical  workforce  includes  recent  or  one-time  end-users 
in  order  to  create  greater  symmetry  between  end  capability  and 
changing  user  needs 

•  SoS-level  technical  workforce  is  small  and  agile  to  avoid  parochial 
interests 

•  Lean  but  dedicated  programmatic  workforce  to  create  and  support 
technical  expertise 

Incentive  structure 

•  Uses  budgetary  and  personnel  levers  to  foster  mission  commitment  and 
compliance  with  established  standards 

•  Low  barriers  to  the  entry  of  new  technologies  and  the  innovation  of 
existing  solutions 

Knowledge  ownership/ 
access  to  knowledge 

•  Leadership  shows  sensitivity  to  the  wide  range  of  information-handling 
capacity  and  ability  of  different  stakeholders  throughout  an  organization 
to  manage  and  understand  information 

•  Leadership  also  campaigns  to  accommodate  those  stakeholder  entities 
with  lower  information-handling  capacity  and  ensure  their  interests  are 
also  incorporated  into  the  information  feedback  loop 

•  Standards  and  network  backbones  are  freely  shared  amongst 
stakeholders  to  encourage  user-level  innovation  and  collaboration 

Risk  assessment/risk 
management 

•  Critical  technologies  are  highly  mature  and  commercial  off-the-shelf 
(COTS)  where  possible 

•  Clear,  measurable  metrics  are  established  to  monitor  less  mature 
technologies  and  changes  in  the  external  environment 

Concluding  Thoughts  and  Topics  for  Further  Research 

The  research  and  analysis  presented  in  this  abridged  version  and  explored  in  the 
associated  CSIS  technical  report  represent  an  important  contribution  to  DoD  efforts  to 
improve  its  performance  in  delivering  large,  capable  defense  systems.  This  is  a  critical  need 
for  the  nation’s  defense  planners  and  acquisition  leaders.  Downward  budget  pressure  is 
likely  to  continue  into  the  foreseeable  future,  and  the  threats  are  becoming  more  complex 
due  to  the  increasing  pace  of  global  technological  change  and  democratization  of  the  means 
to  inflict  harm.  Both  of  these  pressures  demand  more  robust  governance  for  SoS 
acquisition. 

Additional  research  is  necessary  to  refine  the  proposed  governance  model,  test  its 
theoretical  impact  on  acquisition  efforts  through  desktop  systems  modeling  and  simulation 
exercises,  and  observe  its  applicability  to  other  challenges  related  to  complexity. 

Subsequent  research  might  attempt  to  answer  the  following  pressing  topics: 
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•  Can  exceptional  performance  in  enabling  attributes  make  up  for  poor 
performance  in  the  three  critical  attributes? 

•  What  human  capital  planning  helps  ensure  success  in  the  three  critical 
attributes,  especially  decision-making  authority? 

•  How  can  the  governance  framework  be  applied  to  other  complex  problems, 
such  as  battlefield  command  and  corporate  governance? 

•  What  quantitative  modeling  methods  can  be  used  to  predict  attribute 
shortfalls  and  simulate  acquisition  success  and  failure  scenarios? 


These  topics  represent  only  a  few  areas  of  interest  for  further  research.  Additional  research 
would  apply  successes  in  complexity  research  from  other  disciplines  to  the  challenge  of 
mapping  complex  acquisition  efforts  and  forecasting  mission  success. 
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